Convolutional Neural Networks as Feature Generators for Near-duplicate Video Detection

With ever increasing online repositories of video content, accurate and fast near-duplicate detection is important; it can help deter piracy and distribution of illegal content, while improving user experience when searching existing video collections by de-duplicating results and providing contextual data, such as links to similar videos or helping to extract meta data by clustering content. Existing methods are a trade-o↵ between frame-level accuracy, using complex spatial annotations as di↵erences of gaussians, and computational eciency, using video-level annotation of color and motion. With the increasing power and feasibility of deep neural networks, they provide a potential middle-ground, by providing a per-frame signature in terms of the dominant classification while still being relatively performant. In this implementation, pre-trained networks from the Ca↵e project, including AlexNet, GoogleNet and the R-CNN networks are used to construct a searchable database of video signatures, that can be queried by looking at the intersection of dominant features across frames. Using an evaluation dataset of approximately 250 videos, the R-CNN network is shown to produce a set of features that is resilient to common distortions, such as small rotations, cropping, re-encoding and changes to color and brightness. ROC curves show the network has a high degree of accuracy at matching known videos and distorted videos, while rejecting novel content. These results suggest that neural network-based near-duplicate detection is both feasible and accurate. Yet the current implementation is constrained by inherent limitations in the pre-trained networks, which were trained on a limited number of labels and are sensitive to strong color or rotational changes. Future work will look at training a novel neural network, as well as adapting the feature database to do similar-video search and meta-data extraction.